Skip to content

Missing pv.netcdf - #343

Open
Sharkyii wants to merge 9 commits into
openclimatefix:mainfrom
Sharkyii:fix-test
Open

Missing pv.netcdf#343
Sharkyii wants to merge 9 commits into
openclimatefix:mainfrom
Sharkyii:fix-test

Conversation

@Sharkyii

@Sharkyii Sharkyii commented Feb 14, 2026

Copy link
Copy Markdown
Contributor

Description

This PR fixes the evaluation script which was broken due to changes in the Hugging Face dataset structure. The dataset no longer provides a single pv.netcdf file but instead uses partitioned Parquet files organized by year and month (5_minutely/year=YYYY/month=MM/data.parquet).

Changes:

  • Updated get_pv_truth() in quartz_solar_forecast/eval/pv.py to download and process monthly Parquet files instead of a single NetCDF file
  • Replaced xarray dependency with pandas for Parquet handling
  • Added smart caching to only download required year-month combinations based on testset
  • Updated column name mappings to match new schema (ss_idpv_id, datetime_GMTtimestamp, generation_Whgeneration_wh)
  • Added timezone handling (UTC localization) to match PV data timestamps
  • Added error handling for corrupted Parquet files (auto-delete and skip)
  • Updated .gitignore to exclude temporary directories

Dependencies:

  • Requires pyarrow for Parquet file support

Fixes #283

How Has This Been Tested?

Tested with the full evaluation pipeline

@Sharkyii

Copy link
Copy Markdown
Contributor Author

@peterdudfield could you review this when you have time?

@Sharkyii

Copy link
Copy Markdown
Contributor Author

@peterdudfield this is ready to merge?

@peterdudfield

Copy link
Copy Markdown
Contributor

Hye, Would you be able to get unit tests working? Or is that something else?

@Sharkyii

Copy link
Copy Markdown
Contributor Author

Test: tests/unit/api_tests/v0/test_api.py::test_api_ok

Root error:

NameError: name 'RequestsCookieJar' is not defined

Where it happens:
Deep in requests_cache's cattrs serializer, when trying to cache an HTTP response from the Open-Meteo weather API.

The chain:

  1. Test calls POST /forecast/
  2. API calls get_nwp() → hits Open-Meteo via openmeteo-requests
  3. openmeteo-requests uses requests-cache 1.2.0 to cache the response
  4. requests-cache uses cattrs to serialize the response
  5. cattrs 26.1.0 tries to resolve type hints on a requests internal class
  6. requests 2.34.2 uses a forward reference 'RequestsCookieJar' in its type annotations
  7. cattrs can't resolve that forward reference → NameError
  • Downgrading cattrs to 23.2.3 didn't help — same error persists
  • Downgrading requests to 2.31.0 was suggested but the venv has other packages (scipy, xgboost) compiled against newer numpy/requests ABIs, causing binary incompatibilities
  • The project's pyproject.toml doesn't pin cattrs or requests at all, so any fresh install pulls incompatible versions

@peterdudfield have a look

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test failing for eval

2 participants